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The  goal  of  this  MURI  program  (ACCLIMATE)  has  been  to  develop  adaptive,  intelligent,  and  multi¬ 
agent  control  technologies  to  address  the  increasing  concerns  on  the  reliability,  adaptability,  and  robustness 
of  complex  systems  (such  as  unmanned  air/ground  vehicles)  encountered  on  modern  battlefields  or  in  home¬ 
land  defense.  In  particular,  the  program  has  been  focused  on  the  following  three  thrusts: 

1.  Architecture  design  of  adaptive  heterogeneous  systems.  We  design  novel  system  architectures  that 
organically  incorporate  human  intervention,  and  can  be  composed  interoperably  on  the  fly. 

2.  Infegrafion  of  mulli-sensor  informalion  info  3-D  virfual  environmenfs  for  incorporafing  human  infer- 
venfion. 

3.  Handling  uncerfainly  and  adversarial  infenf  in  adapfive  planning. 

In  fhis  final  reporf,  we  summarize  and  highlighf  fhe  key  confribufions  of  fhis  program  fowards  achieving 
fhe  above  fundamenlal  goals. 

1  Thrust  I:  Architecture  Design  of  Adaptive  Heterogeneous  Systems. 

1.  A  llighl  confrol  sysfem  for  aerial  robofs. 

We  have  proposed  a  hierarchical  flighf  confrol  system  for  unmanned  aerial  vehicles  [4].  The  proposed 
sysfem  execufes  high-level  mission  objecfives  by  progressively  subslanfiafing  fhem  info  machine-level 
commands.  The  acquired  informalion  from  various  sensors  is  propagaled  back  lo  fhe  higher  layers 
for  reacfive  decision  making.  Each  vehicle  is  connected  via  sfandardized  wireless  communication 
protocol  for  scalable  multi-agent  coordination. 

Eigure  1  shows  the  overall  layers  in  the  hierarchical  flight  management  system.  Eirst,  motion-related 
information  that  is  vital  for  vehicle  control  and  high-level  operation  is  measured  by  the  onboard  nav¬ 
igation  sensors  in  the  sensing  layer.  These  mobile  sensors  include  inertial  navigation  system  (INS), 
global  positioning  system  (GPS),  ultrasonic  sensors,  and  laser  trange-finders.  Second,  the  reasoning 
and  coordination  layer  follows.  Each  component  in  this  layer  is  responsible  for  strategy  planning  of 
a  specific  fask.  Any  appropriate  sfrafegy  planner  for  a  given  mission  is  fhen  selecfed  by  a  swifching 
layer.  Einally,  fhe  action  layer  insfrucfs  fhe  UAV  fo  move  fo  sfrafegic  locafions  fhaf  are  compufed  by 
fhe  decision  making  process. 

The  proposed  sysfem  has  been  successfully  implemented  on  a  number  of  small  helicopfers  and  vali- 
dafed  in  various  applications,  including  waypoint  navigation,  pursuit-evasion  game,  and  target  track¬ 
ing. 

2.  Pursuif-evasion  sfrafegies  for  teams  of  mulfiple  agenfs  wifh  incomplefe  informalion. 

We  have  investigated  search  sfrafegies  for  mulli-player  pursuif-evasion  games,  in  which  a  learn  of 
pursuers  fry  lo  delecl  and  caplure  mulfiple  inlelligenl  evaders  [1].  These  are  games  wifh  incomplete 
informalion  for  which  Ihere  is  no  complele  Iheory  of  exislence  of  optimal  solutions.  In  fhe  pursuif- 
evasion  game  fhaf  we  consider,  a  pursuer  (or  a  group  of  pursuers)  searches  for  an  evader  (or  a  group 
of  evaders).  A  suifable  physical  analogy  would  be  a  mililary  surveillance  mission  of  a  public  safely 
operalion  wifh  unmanned  aerial  vehicles  (UAVs)  and  unmanned  ground  vehicles  (UGVs)  serving  as 
fhe  pursuers  and  evaders,  respeclively.  In  our  selling,  all  fhe  players  acl  in  a  bounded  reclangular 
area.  The  pursuers  can  sense  only  a  finite  region  around  fhem,  which  presenls  fhe  difficully  of  parlial 
observabilily.  Based  on  fhe  history  of  such  limifed  sensing  informalion,  fhe  pursuers  fry  oulsmarl  fhe 
evaders  and  evenfually  caplure  fhem. 
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Figure  1 :  The  multi-funetional  hierarehieal  flight  management  system. 


In  this  work,  we  first  ineorporated  overlap-redueing  algorithms  to  enable  multiple  pursuers  to  ehase 
multiple  evaders.  These  new  algorithms  reduee  the  amount  of  sensing  overlap  between  pursuers,  thus 
distributing  them  more  effieiently.  Furthermore,  we  developed  heuristie  algorithms  and  polieies  for 
pursuers  with  varying  sensing  eapabilities  against  faster  evaders  with  enhaneed  intelligenee. 

The  effeetiveness  of  the  proposed  pursuit  polieies  was  evaluated  via  simulation.  The  results  suggested 
that  the  globa- subregional  and  the  local-max  polieies  perform  the  best  among  the  five  tested  pursuit 
polieies.  The  overall  better  performanee  of  these  two  polieies  ean  be  attributed  to  two  faetors:  1.  The 
reduee  sensing  overlap  between  pursuers.  2.  They  do  not  over-assign  pursuers  to  target  loeations.  Our 
analysis  also  provides  a  basis  for  developing  poliey  elasses  for  the  study  of  finite  horizon  games. 

3.  Synehronization  in  oseillator  networks. 

Understanding  how  synehronization  is  aehieved  in  a  network  provides  us  with  invaluable  information 
for  eonstrueting  large-seale  systems.  For  example,  in  leaderless  eoordination  of  multi-agent  systems, 
a  key  issue  is  how  to  aehieve  eonsensus  for  arbitrary  topologies.  Perhaps  the  most  eelebrated  model 
for  synehronization  is  the  Kuramoto  model,  a  system  of  struetured  ordinary  differential  equations, 
whieh  has  been  used  to  explain  how  synehronization  is  aehieved  in  many  engineering,  physieal,  and 
biologieal  systems. 

In  our  work  [11],  we  have  eonsidered  the  problem  of  synehronization  of  eoupled  oscillator  networks 
in  the  Kuramoto  framework  with  heterogeneous  communication  delays.  We  have  manifested  that 
synchronization  can  be  achieved  even  if  the  oscillators  are  non-identical,  if  we  allow  heterogeneous 
delays  in  the  networks.  Moreover,  we  have  established  synchronization  for  Kuramoto  network  with 
switching  topologies  even  in  the  presence  of  delays. 

We  have  shown  that  the  synchronization  of  the  delayed  Kuramoto  models  may  not  be  global  in  the 
nonlinear  case,  as  in  general  the  solution  of  the  consistency  may  not  be  unique.  A  nonlinear  result 
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similar  to  the  one  obtained  in  the  case  of  network  congestion  control  would  be  desirable,  at  least  in 
the  case  in  which  there  is  a  unique  attracting  set.  This  will  be  the  focus  of  our  future  research. 

4.  On  consensus  over  random  networks. 

Over  the  past  few  years,  algorithms  for  solving  different  variants  of  the  decentralized  consensus  prob¬ 
lem  have  received  a  significant  amount  of  attention.  In  general,  the  main  goal  of  such  algorithms  is 
to  achieve  a  global  objective  over  a  network  of  agents  using  purely  local  interactions.  Due  to  absence 
of  a  centralized  computational  entity,  possible  lack  of  information  on  global  topology  and  limited  en¬ 
ergy  resources,  one  desirable  property  of  such  algorithms  is  robustness  to  occurrence  of  discontinuous 
changes  in  the  topology  of  the  multiagent  network.  These  topology  changes  may  occur  naturally  due 
to  communication  constraints  and/or  movement  of  agents. 

Our  project  [12]  presented  a  more  general  framework,  compared  to  the  already  existing  results,  for 
almost  sure  convergence  of  the  agreement  protocols  for  a  randomly  changing  network.  Our  framework 
contains  both  the  random  gossip  algorithm  with  synchronous  time  model  and  the  Erdos-Renyi  random 
graph  model  as  special  cases.  Moreover,  the  results  are  more  general  comparing  to  those  that  only 
provide  a  sufficient  condition  for  reaching  consensus  in  probability,  while  our  theorems  provide  a 
necessary  and  sufficient  condition  for  almost  sure  convergence. 

The  main  assumption  behind  our  model  is  that  the  underlying  random  graphs  of  the  network  are 
mutually  independent  over  time  and  have  the  same  distribution.  We  also  assume  that  the  parameters 
of  the  distributed  consensus  algorithm  at  any  given  time  step  are  independent  of  the  parameters  at  all 
other  times  and  are  drawn  from  the  same  probability  distribution.  In  the  future,  we  plan  to  extend  our 
results  for  the  more  general  case  of  dependence  over  time  to  include  the  case  of  agents  with  motion. 


2  Thrust  II:  Integration  of  Multi-Sensor  Information  into  3-D  Virtual  En¬ 
vironments  for  Incorporating  Human  Intervention. 

1 .  Wide  area  multiple  camera  calibration. 

The  calibration  of  cameras  distributed  in  a  wide  area  is  a  challenging  task  because  it  is  impossible  to 
use  reference  objects  visible  to  all  cameras  and  because  wide  field-of-view  cameras  suffer  under  radial 
distortion.  Our  work  [2]  proposes  the  first  algorithm  in  the  literature  for  radial  distortion  estimation 
from  multiple  views  without  involving  non-linear  minimization.  The  correspondences  between  views 
are  obtained  by  deliberately  moving  an  LED  in  thousands  of  unknown  positions  in  front  of  the  cam¬ 
eras.  Then  both  projection  matrices  and  radial  distortion  parameters  are  simultaneously  computed 
using  a  factorization  approach. 

The  main  contribution  of  this  paper  is  that  the  simultaneous  treatment  of  projection  matrices  and  radial 
distortion  over  multiple  views  and  without  any  nonlinear  minimization.  The  algorithm  differs  from 
others  in  the  literature  for  being  a  one-shot  method  with  minimum  computational  effort.  Moreover, 
it  makes  use  of  all  multiple-view  constraints  simultaneously,  instead  of  just  pairwise  fundamental 
matrices.  Compared  with  the  existing  methods,  it  provides  an  estimation  for  the  projection  matrices 
and  radial  distortion  without  requiring  Euclidean  stratification  and  nonlinear  minimization. 

2.  Experiments  on  visual  loop  closing  using  vocabulary  trees. 

Imagine  a  robot  vehicle  traveling  in  an  urban  environment  with  a  camera.  The  goal  of  visual  loop 
closing  is  to  detect  when  the  robot  is  close  to  a  previously  visited  location  based  on  the  images  ac¬ 
quired  previously.  Since  we  do  not  assume  that  any  image  of  the  environment  is  previously  available 
before  the  current  acquisition,  the  whole  process  must  be  performed  in  an  online  fashion.  Hence, 
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to  attain  reasonable  eomputational  time,  it  is  imperative  to  avoid  eomparing  the  eurrent  image  to  all 
previously  seen  images.  In  our  experiments,  we  used  GPS  positioning  as  a  prior. 

In  this  work  [5],  we  studied  the  problem  of  visual  loop  elosing  for  long  trajeetories  in  an  urban 
environment.  Our  approaeh  is  based  on  the  voeabulary  tree  method,  whieh  is  built  to  define  a  set 
of  visual  words.  Features  are  said  to  be  similar  if  they  eorrespond  to  the  same  visual  word.  The 
proeedure  can  be  summarized  as  follows.  A  vocabulary  tree  is  built  offline.  Image  descriptors  from 
each  image  of  the  sequence  are  added  sequentially  to  the  vocabulary  tree  using  inverted  files.  When 
the  vehicle  re-approaches  a  previously  visited  location,  the  descriptors  of  the  current  image  are  used 
in  conjunction  with  the  inverted  files  to  obtain  the  N  closest  images.  Then,  geometric  consistency 
using  the  epipolar  constraint  is  used  to  determine  if  the  matches  are  good.  We  experiment  with  two 
types  of  trees  and  with  features  from  the  trajectory  as  well  as  with  features  completely  independent  of 
the  trajectory. 

3.  Digitizing  archaeological  excavations  from  multiple  views. 

We  proposed  a  novel  approach  on  digitizing  large  scale  unstructured  environments  like  archaeological 
excavations  using  off-the-shelf  digital  still  cameras  [14].  The  cameras  are  calibrated  with  respect  to 
few  markers  captured  by  a  theodolite  system.  Our  algorithm  has  as  input  multiple  calibrated  images 
and  outputs  an  occupancy  voxel  space  where  occupied  pixels  have  a  local  orientation  and  a  confidence 
value.  Both  orientation  and  confidence  facilitate  an  efficient  rendering  and  texture  mapping  of  the 
resulting  point  cloud. 

The  algorithm  combines  the  following  new  features:  Images  are  backprojected  to  hypothesized  local 
patches  in  the  world  and  correlated  on  these  patches  yielding  the  best  orientation.  Adjacent  cameras 
build  tuples  which  yield  a  product  of  pairwise  correlations,  called  strength.  Multiple  camera  tuples 
compete  each  other  for  the  best  strength  for  each  voxel.  A  voxel  is  regarded  as  occupied  if  strength 
is  maximum  along  the  normal.  Unlike  other  multi-camera  algorithms  using  silhouettes,  photocon¬ 
sistency,  or  global  correspondence,  our  algorithm  makes  no  assumption  on  camera  locations  being 
outside  the  convex  hull  of  the  scene.  Figure  2  demonstrates  a  reconstruction  sequence. 

4.  Correspondence-free  structure-from-motion. 

Estimation  of  3-D  motion  from  two  calibrated  views  has  been  exhaustively  studied  in  the  case  where 
optical  flow  or  feature  correspondences  are  given  and  the  scene  is  rigid.  Algorithms  working  over 
multiple  frames  yield  high-quality  motion  trajectories  and  reconstructions  where  feature  matches  are 
cleaned  through  outlier  rejection  and  motions  independent  of  the  camera  are  excluded.  These  outlier 
rejection  and  segmentation  steps  are  subject  to  fundamental  coupling  of  data  association  and  estima¬ 
tion:  If  we  knew  the  motion  estimate,  data  association  would  be  trivial;  if  we  knew  the  data  associa¬ 
tion,  motion  estimation  would  be  easier.  Resistance  to  outliers  and  independent  motions  pose  severe 
practical  limitations  to  the  wide  application  of  structure  from  motion  as  a  navigation  tool,  visual  GPS, 
or  a  camera  tracker. 

we  have  proposed  a  novel  approach  for  structure-from-motion  applicable  in  the  presence  of  large 
motions  and  many  irrelevant  features  resulting  from  reduced  overlap  of  the  helds  of  view  [9].  The 
approach  is  based  on  the  naive  principle  that  an  exhaustive  search  over  all  possible  correspondence 
conhgurations  for  all  motion  hypotheses  would  yield  all  3-D  motions  compatible  with  these  two 
views.  Such  a  search  is  nevertheless  intractable  when  we  use  a  large  held  of  view.  The  contribution  of 
this  paper  is  in  the  reformulation  of  this  Hough-reminiscent  approach  as  a  hltering  problem:  Assuming 
a  similarity  function  between  any  two  features  in  the  hrst  and  second  view,  we  convolve  this  function 
with  a  kernel  that  checks  the  compatibility  of  a  correspondence  pair  with  the  epipolar  constraint 
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(a)  Two  original  image  pairs. 


(b)  Reconstruction  of  the  scene. 

Figure  2:  Illustration  of  a  reconstruction  sequence. 


for  a  given  motion  hypothesis.  The  resulting  integral  is  a  Radon  transform  known  from  computer 
tomography  where  a  material  density  is  integrated  over  a  ray  path. 

We  present  a  complete  end-to-end  system,  from  images  to  motion  parameters  where  the  only  tuning 
parameter  is  the  coupled  resolution  of  the  image  and  the  motion  space.  We  extract  SIFT  features  for 
which  we  define  their  similarity  function  proportional  to  the  Euclidean  norm  of  the  attribute  vectors 
and  we  compute  the  spherical  harmonics  of  the  similarity  function  as  the  input  to  the  correlation 
integral.  The  results  on  real  sequences  compare  favorably  to  traditional  robust  estimation  methods. 

5.  Full  automatic  registration  of  3-D  point  clouds.  During  the  last  few  years  we  have  experienced  the 
market  introduction  of  range  sensors  with  reasonable  cost  as  well  as  the  availability  of  many  successful 
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stereo  vision  algorithms.  As  any  registration  problem,  range  registration  consists  of  the  steps  of 
matching  and  estimation  of  the  rigid  transformation.  Depending  on  the  displacement  and  orientation 
between  point  clouds,  we  differentiate  between  crude  and  fine  alignment.  The  challenge  in  crude 
registration  lies  in  performing  it  automatically  and  consistently  even  when  there  is  very  small  overlap. 
The  best  existing  technique  for  fine  regisfrafion  is  fhe  Iferafive  Closesf  Poinf  (ICP)  algorifhm. 

We  have  proposed  a  novel  fechnique  for  fhe  regisfrafion  of  3-D  poinf  clouds  which  makes  very  few 
assumptions  [8].  We  avoid  any  manual  rough  alignmenf  or  fhe  use  of  landmarks,  displacemenf  can 
be  arbifrarily  large,  and  fhe  fwo  poinf  sefs  can  have  very  liffle  overlap.  Crude  alignmenf  is  achieved 
by  esfimafion  of  fhe  3-D  rofafion  from  fwo  Exfended  Gaussian  Images  (EGIs)  even  when  fhe  dafa 
sefs  inducing  fhem  have  partial  overlap.  The  technique  is  based  on  fhe  correlafion  of  fhe  fwo  EGIs 
in  fhe  Eourier  domain  and  makes  use  of  fhe  spherical  and  rofafional  harmonic  Iransforms.  Eor  pairs 
wifh  low  overlap  which  fail  a  crifical  verificafion  sfep,  fhe  rofafional  alignmenf  can  be  obfained  by  fhe 
alignmenf  of  constellation  images  generated  from  fhe  EGIs.  Rofafionally  aligned  sefs  are  mafched  by 
correlafion  using  fhe  Eourier  fransform  of  volumefric  funcfions.  A  fine  alignmenf  is  acquired  in  fhe 
final  sfep  by  running  Iferafive  Closesf  Poinfs  wifh  jusf  a  few  iterations. 
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Eigure  3:  (A)  A  represenfafive  room  range  scan.  (B)  Poor  alignmenf  obfained  by  running  ICP  on  fhe  inpuf. 
(C,D)  A  side  and  overhead  view  of  fhe  rough  alignmenf.  (E,P)  A  full  and  partial  view  of  fhe  final  alignmenf. 


6.  Single-viewpoinf,  cafadiopfric  cone  mirror  omnidirectional  imaging  fheory  and  analysis. 

Ordinary  cameras  used  in  machine  vision  eifher  have  a  narrow  field  of  view  (EOV)  or  have  a  wide 
EOV  buf  suffer  from  complex  disforfion.  If  can  be  difficulf  fo  unwarp  a  wide  EOV  image  fo  obfain 
perspecfive  projection  views  accurafely.  Based  purely  on  fhe  ideal  projection  imaging  model,  if  has 
been  shown  fhaf  surfaces  of  revolufion  of  conic  section  curves  are  fhe  only  mirror  shapes  fhaf  can  be 
paired  wifh  a  single  converging  projecfion  camera  fo  creafe  single-viewpoinf  (SVP),  cafadiopfric  om- 
nidirecfional  view  sysfems  whose  omniview  image  can  be  unwarped  fo  perspecfive  projecfion  views 
wifhouf  sysfemafic  disforfions.  The  pinhole-model-based  geomefry  has  also  been  analyzed  by  ofhers. 
The  condifion  critical  fo  being  able  fo  unwarp  fo  perspecfive  projecfion  views  from  a  single  omniview 
image  is  fhe  SVP  condifion.  The  cone  shape,  alfhough  a  surface  of  revolufion  of  a  conic  secfion,  has 
been  deemed  unusable  because,  if  had  been  said,  in  ifs  SVP  configuration  only  rays  grazing  fhe  mirror 
surface  can  be  seen.  In  facl,  practical  SVP  cone  mirror  omniview  sysfems  can  be  consfrucfed;  fhis 
work  [7]  is  an  expansion  on  fhaf  imporfanf  discovery.  We  show  fhaf  even  under  fhe  pinhole  camera 
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model  the  SVP  cone  configuration  works  and  sees  any  world  point  in  its  FOV,  not  just  rays  grazing 
the  mirror  surface. 

The  cone  mirror  has  not  been  used  to  construct  an  SVP  omnidirectional  imaging  sensor  that  can 
reproduce  perspective  projection  views  from  a  single  omniview  image  before  our  work.  However, 
cone  mirrors  have  been  used  to  aid  navigation,  map  building,  collision  avoidance,  and  pipe  inspection 
in  non-SVP  configurations.  The  cone  mirror  images  were  used  as  is,  and  no  attempt  was  made  to 
unwarp  them  to  undistorted  images.  By  using  multiple  normal  cameras  positioned  properly  in  relation 
to  a  plane  mirror  pyramid,  a  high-resolution,  SVP,  wide-FOV  system  can  be  built.  The  trade-offs, 
though,  are  the  high  price  and  complexities  involved  with  multiple  cameras.  Bulky  size,  weight, 
calibration,  synchronization,  and  gain  differences  are  problems  associated  with  multicamera  systems 
that  single-camera  systems  are  free  of. 

An  SVP  system  is  worthwhile  if  the  benefits  outweigh  the  drawbacks  for  a  particular  application. 
Only  with  SVP  can  a  catadioptric  omnicam  use  a  single  range-independent  lookup  table  or  formula 
for  correct  unwarping.  The  SVP  cone  system  is  cheap  and  simple  to  build,  operate,  and  maintain  while 
retaining  a  decent  vertical  resolution  and  good  flexibility  in  SVP.  The  SVP  cone  system  is  therefore 
always  worth  evaluating  before  considering  more  complex  and  expensive  omniview  sensors.  The 
main  purpose  of  our  work  here  is  to  prove  that  an  SVP  cone  system  is  both  theoretically  and  physically 
viable  and  to  present  a  detailed  analysis  of  cone  SVP  systems  that  provides  systematic  physics-based 
guidelines  for  deciding  whether  the  SVP  cone  is  suitable  for  a  particular  application. 

We  analyze  here  the  aberrations  of  SVP  cone  mirror  systems  using  accurate  numerical  optical  ray 
tracing.  With  our  analysis  we  derive  an  optical  design  that  minimizes  such  aberrations.  The  cone  is 
among  the  simplest  mirror  shapes  to  produce,  and  it  has  much  higher  meridional  (tangential)  angular 
resolution  compared  with  other  conic  section  mirrors  for  scenes  around  the  horizon.  It  adds  the  least 
optical  distortion  to  the  resulting  meridional  images  because  it  is  the  only  omniview  mirror  with  a 
noncurved  mirror  surface  in  the  meridional  cross  sections. 


3  Thrust  III:  Handling  Uncertainty  and  Adversarial  Intent  in  Adaptive  Plan¬ 
ning. 

1.  Decentralized  nonlinear  model  predictive  control  of  multiple  flying  robots. 

We  have  studied  nonlinear  model  predictive  control  (NMPC)  for  multiple  autonomous  helicopters 
in  a  complex  environment.  NMPC  has  been  recognized  as  a  suitable  framework  for  the  control  of 
nonlinear  dynamic  systems  subject  to  operating  contraints.  The  inherent  expandability  of  NMPC, 
which  can  include  various  performance  and  constraints  as  a  cost  function,  has  shown  promise  for  the 
control  of  unmanned  vehicles.  However,  the  computational  load  of  the  conventional  NMPC  was  often 
prohibitive  for  many  dynamic  systems  with  a  fast  response. 

In  our  work  [13],  we  proposed  to  resolve  the  path  planning  and  optimal  control  problems  for  multiple 
mobile  robots  in  a  complex  3-D  environment  by  nonlinear  model  predictive  control  and  potential 
function  techniques.  We  have  shown  that  our  integrated  approach  can  solve  various  realistic  scenarios 
involving  multiple  UAVs  and  obstacles  in  a  complex  3-D  environment. 

The  proposed  nonlinear  trajectory  planning  and  tracking  controller  has  been  evaluated  in  several  rep¬ 
resentative  scenarios.  Two  examples  are  shown  in  Figure  4  and  5,  respectively. 

2.  Efficient  gradient  estimation  for  motor  control  learning. 


Figure  4:  Distributed  collision  avoidance:  (a)  initial  configuration  and  the  destination  of  each  helicopter, 
and  the  corresponding  desired  trajectories;  (b)  their  trajectories  during  execution. 


-30 


Figure  5:  Pursuit-evasion  in  a  3-D  environment:  lines  between  the  pursuer  (red)  and  the  evader  (blue) 
represent  the  corresponding  positions  at  each  time  instant. 


The  task  of  estimating  the  gradient  of  a  function  in  the  presence  of  noise  is  central  to  several  forms 
of  reinforcement  learning,  including  policy  search  methods.  We  present  two  techniques  for  reducing 


9 


gradient  estimation  errors  in  the  presenee  of  observable  input  noise  applied  to  the  eontrol  signal  [6]. 
the  first  method  extends  the  idea  of  a  reinforeement  baseline  by  fitting  a  loeal  model  to  the  response 
funetion  whose  gradient  is  being  estimated;  we  shown  how  to  find  fhe  response  surfaee  model  fhaf 
minimizes  fhe  varianee  of  fhe  gradienf  esfimafion,  and  how  fo  esfimafe  fhe  model  from  dafa.  The 
seeond  mefhod  improves  fhis  furlher  by  diseounfing  eomponenfs  of  fhe  gradienf  veefor  fhaf  have  high 
varianee.  These  mefhods  are  applied  fo  fhe  problem  of  mofor  eonfrol  learning,  where  aefuafor  noise 
has  a  signiheanf  influenee  on  behavior.  In  parfieular,  we  apply  fhe  feehniques  fo  learn  loeally  opfimal 
eonfrollers  for  a  darf-fhrowing  fask  using  a  simulafed  fhree-link  arm;  we  demonsfrafe  fhaf  fhe  proposed 
mefhods  signifieanfly  improve  fhe  response  funefion  gradienf  esfimafe  and,  eonsequenfly,  fhe  learning 
eurve,  over  exisfing  mefhods. 

3.  Solving  models  of  eonfrolled  dynamie  planar  rigid-body  systems  wifh  friefional  eonfaef. 

The  effeefive  operafion  of  robofie  sysfems  in  fhe  real  world  erifieally  depends  on  fheir  reliable  in- 
feraefion  wifh  fhe  environmenf  fhrough  bofh  infenfionally  and  uninfenfionally  esfablished  friefional 
eonfaefs.  Robusf  manipulafion  of  eifher  fhe  robof  or  fhe  environmenf  fhrough  sueh  inferaefions,  espe- 
eially  for  fasks  whieh  depend  explieifly  on  aeeurafe  eonfrol  of  eonfaef  forees,  remains  a  ehallenging 
problem  for  fhe  robofies  eommunify.  The  primary  diffieulfy  arises  from  eomplexifies  assoeiafed  wifh 
eaeh  of  fhe  variefy  of  model  ehoiees  available  for  deseribing  inferaefions  befween  fwo  objeefs  fhaf  are 
in  eonfaef. 

In  our  sfudy,  we  adopf  a  elass  of  models  fhaf  presume  rigid  objeefs  and  Coulomb  frielion  af  eonfaef 
poinf.  If  is  our  eonjeefure  fhaf  fhe  resulting  models,  whieh  sfill  presenting  some  eompufafional  dif- 
fieulfies  due  fo  non-linearify  and  ambiguify,  represenf  an  effeefive  and  fraefable  basis  for  fhe  design 
of  reliable  behavioral  eonfrol  sysfems  for  eonfaef  fasks.  The  main  eonfribufion  is  a  feehnique  and  an 
assoeiafed  algorifhm  whieh  eompufes  fhe  insfanfaneous  dynamie  response  for  eonfrolled  mulfi-rigid- 
body  sysfems  subjeef  fo  Coulomb  friefional  eonfaefs  [3]. 

The  proposed  algorifhm  makes  if  possible  fo  predief  fhe  range  of  possible  insfanfaneous  body  ae- 
eelerafions  and  eonfaef  forees  fhaf  mighf  resulf  from  fhe  applieafion  of  a  speeifie  eonfrol  inpul  al 
a  speeifie  poinf  in  fhe  sfafe  spaee  of  fhe  system.  Our  approaeh  is  fo  eonslruel  a  dynamie  response 
funefion  fhaf  is  a  pieeewise  linear  buf  possibly  ambiguous  forward  model  for  fhe  dynamies  of  fhe 
sysfem,  whieh  ean  be  loeally  inverted  fo  aid  in  fhe  generation  of  eonfrol  sfrafegies.  In  building  Ibis 
funefion,  we  also  rigorously  eharaeferize  well-known  problems  assoeiafed  wifh  rigid  objeel  models 
involving  friefional  eonfaefs,  sueh  as  mulliplieily  and  non-exislenee  of  solulions.  Consequenlly,  we 
are  able  fo  undersfand  when  and  how  fhe  underlying  models  fail,  allowing  eonfrollers  fo  avoid  dif- 
fieulf  sellings  or  fhe  reeognize  when  Ihey  are  unavoidable.  Speeifieally,  in  fhe  presenee  of  possible 
ambiguilies,  if  beeomes  possible  fo  design  eonfrollers  lhan  ean  eifher  aflempl  fo  avoid  inpuls  fhaf 
resulf  in  uneerlain  behavior,  or  ehoose  eonlrols  fhaf  maximize  performanee  in  fhe  worse  ease.  The 
end  resulf  is  a  well-informed  eonfrol  sysfem  eapable  of  regulaling  physieal  inleraelion  wifh  fhe  envi- 
ronmenl  fo  produee  desired  behavior.  The  mullimedia  demonslralion  of  Ibis  work  ean  be  aeeessed  al 
http : //www .ijrr.org/ content s/24_l 1/ abstract/ 911 . html. 

4.  Vision-based  dislribuled  eonfrol  laws  for  molion  eoordinalion  of  nonholonomie  robols. 

The  main  underlying  Iheme  of  eooperalive  eonfrol  of  mulfiple  aufonomous  agenls  is  fo  analyze  and/or 
synlhesize  spalially  dislribuled  eonfrol  arehileelures  fhaf  ean  be  used  for  molion  eoordinalion  of  large 
groups  of  aufonomous  vehieles.  A  key  assumption  implied  in  mosl  sueh  appliealions  is  fhaf  eaeh 
agenl  eommunieales  ils  posilion  and/or  veloeily  information  fo  ils  neighbors. 

Inspired  by  fhe  soeial  aggregalion  phenomena  in  birds  and  fish,  researehers  in  robofies  and  eonfrol 
Iheory  have  been  developing  fools,  mefhods  and  algorilhms  for  dislribuled  molion  eoordinalion  of 
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multi-vehicle  system.  In  this  project,  we  propose  a  set  of  control  laws  for  coordinated  motions  such 
as  parallel  and  circular  formations  for  a  group  of  planar  agents  using  purely  local  interactions  [10]. 
The  derived  control  laws  are  in  terms  of  shape  variables  such  as  the  relative  distances  and  relative 
headings  among  the  agents.  However,  these  parameters  are  not  readily  measurable  using  simple 
and  basic  sensing  capabilities.  This  motivates  the  rewriting  of  the  derived  control  laws  in  terms  of 
biological  measurable  parameters.  Each  agent  is  assumed  to  have  only  monocular  vision  and  capable 
of  measuring  basic  visual  quantities  such  as  bearing  angle,  optical  flow,  and  time-to-collision. 

Verification  of  the  theory  through  multi-robot  experiments  demonstrated  the  effectiveness  of  the 
vision-based  control  laws  to  achieve  different  formations.  We  further  provide  inter-agent  collision 
avoidance  properties  to  the  team  members.  What  we  show  is  that  the  two  tasks  of  formation-keeping 
and  collision-avoidance  can  be  done  with  decoupled  additive  terms  in  the  control  law,  where  the  terms 
for  keeping  parallel  and  circular  formations  depend  only  on  visual  parameters. 

In  our  experiment,  we  use  a  series  of  small  form-factor  robots  call  Scarab  shown  in  Figure  6.  The 
Scarab  is  a  20  x  13.5  x  22.2  cm?  indoor  ground  platform  with  a  mass  of  8  kg.  Figure  7  shows  the 
trajectories  of  five  robots  to  form  a  circular  formation  with  collision  avoidance. 


Figure  6:  The  Scarab  is  a  small  robot  with  a  differential  drive  axle.  FED  markers  are  placed  on  top  of  each 
Scarab  for  pose  estimation. 


(a) 


(b) 


(c) 


(d) 


Figure  7:  Five  Scarabs  form  a  circular  formation  starting  with  a  complete-graph  topology  while  avoiding 
collision,  (a)  f  =  0  sec.  (b)  f  =  8  sec.  (c)  t  =  20  sec.  (d)  At  f  =  36  sec.  The  robots  reach  a  stable  balanced 
configuration  around  a  circle  with  radius  of  Im.  The  dash  lines  show  the  actual  trajectories  of  the  robots  and 
their  connectivity  graph  at  the  times  specified  above. 
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